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Abstract. This paper presents a model which can automatically detect a variety 
of student speech acts as students collaborate within a computer supported 
collaborative learning environment. In addition, an analysis is presented which 
gives substantial insight as to how students’ learning is associated with students’ 
speech acts, knowledge that will significantly influence how this model is 
utilized by running learning software. Within Piagetian theory, the cognitive 
conflict of ideas between students is seen as beneficial for learning. Which sorts 
of interpersonal behaviors lead to most effective learning, however, is open to 
debate, with some researchers arguing that cooperation is most effective and 
others arguing that interpersonal conflict is a natural part of collaborative 
learning. We find that, in fact, interpersonal conflict is associated with positive 
learning, a finding that must be taken into account, in designing interventions 
that rely upon detectors of students’ speech acts in CSCL environments. 


1 Introduction 

In recent years, there has been a substantial increase in the use of Computer Supported 
Collaborative Learning (CSCL) environments to promote learning of key educational 
concepts and skills, in a variety of domains. During collaboration, students engage in a 
wide variety of collaborative and learning behaviors, which impact each students’ 
learning in a variety of ways [5, 14, 19]. Bringing collaboration online creates the 
possibility of automatically detecting differences in students’ collaborative behavior and 
responding automatically to differences in students’ behaviors [9], 

However, adapting appropriately depends on achieving three key sub-goals. First, we 
need to know which collaborative behaviors merit intervention. Second, we have to know 
how to intervene appropriately in those cases. Third, we have to be able to accurately 
detect those behaviors so that we can respond to them. In this paper, we present work that 
makes a contribution to the first and third of these goals. We study the learning associated 
with a set of four theoretically interesting collaborative behaviors, within an ecologically 
valid CSCL environment for fractions. We then present a machine-learned model, 
developed within TagHelper [20], which can accurately distinguish a set of collaborative 
behaviors, and performs especially effectively within categories previously determined to 
be of interest. We conclude with a discussion of how the study findings impact how the 
detectors we have developed should be used, to adapt to students’ behavior during CSCL. 
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1.1 Cognitive and Social Conflict in CSCL 

Within this paper, we focus on cognitive and social conflict within computer supported 
collaborative learning, categories found to have relevant impacts on student learning 
within non-computer-mediated collaborative learning. 

Studies from the 1970’s have shown that cognitive conflict promotes cognitive 
development [11,13,21,23,25], in line with Piaget’s [16] writings on the equilibration 
process. Piaget claimed that one source of progress in the development of knowledge is 
found in the imbalance that forces a subject to seek new equilibrium through assimilation 
and accommodation. 

Many researchers have found results indicating that cognitive conflict and learning 
emerges from the process of collaboration, when students mutually engage to co- 
construct shared knowledge [5,14,19]. In fact, Moshman and Geil [12] and Kruger [10] 
have argued that the conceptualization of cognitive change as either a process of conflict 
or a process of cooperation is a false dichotomy, claiming that productive cognitive 
conflicts take place solely within a cooperative context, and not via competition or 
interpersonal conflict. In Moshman and Geil’s view, productive cognitive conflict does 
not emerge from students arguing in favor of their own views, but from co-constructing a 
consensus solution. Similarly, Howe [7] suggested a separation between types of conflict 
that involves transactive [3] dialogues (e.g. cognitive conflict) and interpersonal conflict 
that involves aggression. These studies argued that these two types of interaction occur in 
distinct groups, depending on students’ gender and temperament. 

However, other studies have provided evidence suggesting that cognitive conflict does 
not solely occur in purely collaborative and consensus-based process. For example, 
Arsenio and Lover [2] and Shantz [22], give evidence that the conflict of ideas often 
leads to interpersonal conflict. 

Hence, it appears to still be an open question whether productive cognitive conflict, and 
the learning that emerges from it, only occurs when students show collaborative 
behaviors, or whether it still occurs in conjunction with interpersonal conflict. 

This question is especially relevant within the context of online collaborative learning. 
Online learning has different affordances than the face-to-face learning settings where 
much prior collaborative learning research has taken place - in particular, online 
collaboration, due to the anonymity potentially afforded, is prone to a very high rate of 
insults, often called “flaming” in the online collaboration literature (e.g. O’Sullivan & 
Flanagin [15]). To investigate these questions, we consider the relationship between 
learning and the different behaviors students display in anonymous online collaborative 
learning, focusing on the insults indicative of interpersonal conflict. Our hypothesis is 
that the cognitive conflict which occurs in online learning is highly likely to produce 
interpersonal conflicts, demonstrated by insults; however the online medium may also 
reduces the social consequences associated with insults, meaning that insults may not 
impede learning gains. 
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2 Analysis of Learning Associated With Speech Acts 
2.1 Methods 

Twenty four sixth-grade students from a suburban elementary school near Pittsburgh, PA, 
participated in this study. The study was conducted in a genuine setting of learning, 
involving authentic learning materials. Because one of the students did not use the chat 
interface during the two lab days, that student was excluded from analysis (that student’s 
partner was still included in the sample because this student used the chat interface to 
discuss the problem solving and complain about his partner’s lack of interactivity), and 
the sample was reduced to 23 students. Each student made, on average, 84.3 utterances, 
for a total of 1940 utterances. 

Each student used a mathematics tutoring program covering problems on fraction 
addition, subtraction, multiplication, and division [9], in collaborative pairs mediated 
through TuTalk. TuTalk [8] is a collaborative problem solving interface that include two 
online panels: a chat, and a collaborative interface for the problem solving built in the 
CTAT authoring tool [1], 



Figure 1. Problem-solving interface [9]. 

The students worked in their school lab computer, in pairs, using TuTalk, with their chat 
dialogues and problem solving contributions (within the interface) logged for later 
analysis. The arrangement of the lab was designed so that the students could not easily 
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talk with their pairs outside of the chat interface, with the identity and the seat location of 
the collaborating pairs hidden from their partners. 

Each student individually took nearly isomorphic pre-tests and post-tests, covering 
knowledge of the material covered in the tutor, during a 30 minute period during on 
separate days from tutor usage. The students collaborated in learning fractions within 
TuTalk within two lab sessions, each lasting 45 minutes. This design enabled us to 
investigate the student’s knowledge gains based on the pre- and post-tests, and to analyze 
students’ collaborative and individual learning behaviors. 

2.2 Analytical Method 

In analyzing these dialogues, we divided student behavior into a selected number of 
categories relevant to our analyses. Cognitive communicative categories were split in 
accordance with Youniss and Damon’s [26] interpretation of Piaget’s views on social 
relations in the individual construction of knowledge, where cognitive conflict and 
knowledge construction can occur either through disagreement, where one student 
perceives a misconception or other error in his partner’s thinking and disagrees, 
attempting to express why it is wrong (called disagree with concept in our coding 
scheme). Within this type of disagreement, a student is arguing in favor of his or her own 
views, an ultimately competitive act. In one example, one student said, “i dont think thats 
the common denominator”. By contrast, a student may also refine their partner’s ideas by 
expressing their perspective on an idea, and informing the partner as to their beliefs 
(called inform belief in our coding scheme), attempting to co-construct a solution - a 
more collaborative manner of expression. One example of this within our corpus is, “the 
common denominator is 54”. 

These two categories are shown in Table 1. The two categories do not form an exhaustive 
list of possible cognitive communicative acts, but are particularly relevant to the analysis 
presented here. A fuller taxonomy of speech acts is given in the first author’s doctoral 
dissertation [17]. 


Table 1. Description of the cognitive communicative categories 


Context 

Category 

Description 

More 

Cooperative 

Inform Belief 

The speaker informs his/her partner about his beliefs 
about the concept 
(the common denominator is 54 ) 

Less 

Cooperative 

Disagree with 
Concept 

The speaker disagrees with his partner’s belief about 
the concept. 

(i dont think thats the common denominator) 
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Table 2 Description of social communicative categories 


Offer 

collaborative 

act 

The speaker offers to do something for his or 
her partner towards the problem-solving 
goals 

i do the botttom now 
(student 2b) 

Insult 

The speaker insults his or her partner by 
calling them an obscene or offensive word. 

you loser ( student 
14b) 


The offer collaborative act and insult categories are social communicative categories 
(Table 2). A student who offers collaborative act offers to do something to forward the 
problem-solving goals, generally without having specifically been asked to do so. As 
such, offer collaborative act is a reflection of the peers’ social collaboration. By contrast, 
an insult reflects interpersonal conflict within the dialogue, and is in contrast to social 
collaborative behavior. 

This set of four categories was coded by the first and sixth authors. Both authors coded a 
subset of 225 utterances made by students during collaborative learning. Cohen’s [4] 
Kappa was 0.80, indicating good inter-rater reliability. Afterwards, the protocol analyses 
were based on the first author’s codes for the entire corpus. We also developed a 
machine-learned model that was able to accurately code these categories, discussed later 
in the paper - however, for this analysis human coding was used, as a tractable gold- 
standard. 

2.3 Results 

We analyzed the correlations between pre- and post-test learning gains and the frequency 
of each category of our coding scheme in each pair’s dialogue. The overall pattern of 
results is shown in Table 3. As can be seen, the number of inform belief speech acts a 
student made or received was not significantly correlated with learning gains, 
respectively t(22)=-0.31, p=0.75, t(22)=-0.10, p=0.92 (all tests reported are two-tailed). 
The number of offer collaborative act speech acts a student made or received also was not 
significantly correlated with learning gains, t(22)=0.00, p=0.99, t(22)=0.64, p=0.52, for a 
two-tailed t-test.) Hence, neither of the two cooperative behaviors coded were associated 
with significantly higher learning gains. 

By contrast, the two non-cooperative behaviors were associated with positive learning - 
but only in the student being non-cooperative. The amount a student disagreed with 
concept was associated with statistically significantly higher learning gains for the 
disagreeing student, r=0.53, t(22)=2.93, p<0.01, but not for their partner, t(22)=-0.38, 
p=0.70. The disagree with concept act has the intention to alter the peer’s reasoning with 
conflicting ideas, but appears to have been more of a marker of the disagreeing student’s 
learning than a learning opportunity for their partner. 
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Interestingly, the amount a student made insults was also associated with significantly 
higher learning gains, r=0.70, t(22)=4.53, p<0.001, but receiving insults from another 
student was not associated with higher learning gains, r=0.26, t(22)=l .26, p=0.21. 

Hence, students who acted in ways that create or indicate interpersonal conflict appeared 
to have higher learning gains in this study. Students who behaved in a more cooperative 
fashion did not appear to have higher gains. However, the mechanism explaining this 
result is not clear. Did students learn more because they allowed cognitive conflict to 
move into interpersonal conflict, or did students engage in interpersonal conflict because 
they had learned more than their partner, and were impatient with them? It is possible, in 
particular, that the anonymity of the online learning system facilitated students who had 
just learned the material in choosing to insult their partner rather than help them. 

Table 3. The relationship between learning gains and different dialogue acts (p values shown). 

Statistically significant results (p<.05) in boldface. 


Context 

Cognitive 

Social 

S 

H 

Category 

S 

H 

Category 

More 

Cooperative 

0.8 

0.9 

inform belief 

0.9 

0.5 

Offer 

collaborative 

help 

Less 

Cooperative 

0.008 

0.2 

disagree with 
concept 

0.0001 

0.2 

insult 


S - Speaker, H - Hearer 


3 Development of Collaboration Behavior Detectors 

Having coded a significant number of utterances, the next step was to determine whether 
it would be possible to develop a machine learned model that could automatically detect 
these four categories. Such a detector could be used to drive automated interventions by 
the CSCL environment. (Possible interventions will be discussed in the discussion 
section). 

These four categories were combined with additional data coded with twenty-eight other 
categories, representing a wide span of possible dialogue acts within collaborative 
learning. The full coding scheme is discussed in detail in the first author’s doctoral 
dissertation [17]. In total, there were 170 utterances coded with the 4 speech acts 
discussed above, and a total of 1940 utterances coded with the full set of 32 speech acts. 

Rose et al’s [20] TagHelper tool kit was used to develop a machine learned model that 
could identify the set of speech acts in students’ utterances. TagHelper provides text 
classification services, designed for use with several languages, and access to a variety of 
metrics for validating model goodness. It also automatically distills features previously 
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found to be useful for linguistic analyses, such as bigrams and the presence of “stop” 
words. We used TagHelper to develop models, and to quantify our success in terms of 
agreement with the hand-coded gold standard corpus. The Naive Bayes classification 
algorithm was selected, and applied to the 32 speech acts on the dialogue data. The 
Kappa [4] statistic was used, in combination with 10-fold cross validation, to assess 
reliability of the model’s coding. Non- stratified cross-validation was used, under the 
assumption that multiple utterances by a single student on a single topic are unlikely to be 
highly correlated to each other (as opposed to other types of behavior, where a student’s 
responses may be more characteristic), especially when all students are discussing the 
same mathematical topics. For instance, terms used to discuss fractions, or to disagree 
about fractions, are likely to be similar between students. 

The model was successful at classifying student utterances. Within the whole set of 32 
speech act categories, kappa was a respectable 0.65. Within the set of four utterances that 
were previously thought to be particularly relevant for modeling and understanding 
learning (two were indeed found to be statistically significantly associated with learning), 
kappa was an excellent 0.91. This was better than our human judges’ degree of 
agreement, suggesting that the model was highly successful. 

4 Discussion and Conclusions 

In the previous section, an automated detector of a variety of speech acts was presented 
and shown to be reasonably effective at distinguishing between a variety of speech acts, 
including the four categories discussed in detail in this paper: inform belief, disagree with 
concept, offer collaborative act, and insult. With Kappa values between 0.65 (all 
categories) and 0.91 (categories discussed in this paper), it seems quite feasible to use the 
model for detecting and responding to different types of speech acts. 

However, using the model to drive appropriate interventions depends on understanding 
the implications of each type of speech act, which leads to a need for analyses such as the 
one presented here. In the analysis in section 2 of this paper, learning gains were 
correlated with speech acts. Understanding this gives us an important first piece in the 
puzzle of deciding how a CSCL system should respond to those acts. Developing a full 
understanding of what prompts different speech acts will help us even further. 

Within this study, learning gains were positively related to interpersonal conflict. Arsenio 
and Lover [2] and Shantz [22] previously found, in face to face collaboration, that the 
conflict of ideas can lead to aggressive behavior, including the types of interpersonal 
conflict observed here. In those studies, the aggressive behavior harmed students’ 
interpersonal relationships. However, the anonymity of communication in our study may 
have enabled students to insult each other with lower inteipersonal cost, eliminating one 
of the negative factors associated with aggressive behavior in collaborative learning. In 
broader usage of such a CSCL system, this anonymity could persist within internet usage 
while not persisting in classroom usage (because over time, students can determine who 
they are collaborating with in a classroom, eliminating anonymity). 
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This does not mean, however, that insults are to be encouraged. Insults are clearly 
perceived as problematic in online communication (e.g. O’Sullivan and Flanagin [15]), 
and may be associated with the abandonment of usage of online learning environments 
(cf. Reinig et al. [18]). 

In general, this work supports the hypothesis that positive cognitive conflict can coincide 
with interpersonal conflict. It is not at all clear from our results that the interpersonal 
conflict had a positive impact on cognitive conflict or learning - for instance, it may have 
been a side-effect of one student’s greater learning, with no positive impact on the other 
student. Studying this issue in richer depth will require a combination of methods, 
including time series analysis on a significantly larger corpus of data, and perhaps 
experimentally manipulating interpersonal conflict via not transmitting students’ insults, 
in order to determine insults’ causes and impacts on learning. 

One clear implication of our results is that insults and interpersonal conflict play a 
prominent role in collaborative learning, which cannot be safely ignored. An overly harsh 
response to student insults may also interfere with the positive learning that insults appear 
to be associated with. One approach may be to attempt to develop designs which guide 
students in moderating their comments to others, without disrupting the cognitive conflict 
which insults appear to be associated with. However, if insulting another student 
produces pleasure for the insulting student and increases the insulting student’s desire to 
persist in the use of a learning environment [cf. 24], it may be feasible for a CSCL 
environment to automatically strip out insults from the text the insulted student actually 
receives. Further research on how software that supports CSCL can optimally handle 
insults and other interpersonal conflict behaviors, given the ability to detect those acts, 
appears to be warranted. 
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